AI Still Lags Behind Engineers in Managing IT Outages, Study Shows

AI tools are advancing rapidly in monitoring IT systems, but human engineers remain superior in handling complex outages. A study by Datadog and Carnegie Mellon University analyzed 63 real incidents and over 5 million data points, revealing that while GPT-5 leads generalist models with 62.7% accuracy, human experts achieve 72.7%.

The research highlights current limitations of AI in critical, unforeseen situations. Despite progress in anomaly detection and root cause analysis, automation falls short compared to human problem-solving. The benchmark ARFBench underscores these gaps, particularly in high-stakes production environments.

Collaboration between AI and human specialists could enhance incident response. For now, engineers retain the edge—blending technical intuition with adaptive reasoning that machines cannot yet replicate.